Improving Load Balance and Query Throughput of Distributed IR Systems

As the number of queries grows over time it becomes necessary that Information Retrieval (IR) system provides high query processing rate i.e. high query throughput. In IR systems, there are three types of data partitioning, namely term-based, document-based, and hybrid partitioning. In document-based and hybrid partitioning, query is sent to all nodes and thus high level of parallelism is achieved but low query throughput. In term-based partitioning, a given query is divided into sub-queries and each sub-query is directed to the relevant node. This provides high query throughput and concurrency but poor parallelism and load balance. In this paper, the Moderate Distributed IR System (MDIRS) is proposed to improve the query throughput and load balance of hybrid partitioning. MDIRS inherits the advantage of documentbased partitioning i.e. it provides moderate level of parallelism and the advantage of term-based partitioning. In other words, it provides moderate level of query throughput and load balance. Results from this paper showed that the MDIRS improved the query throughput and the total query response time of hybrid partitioning by 64% over the baseline system.